Systems, devices, and methods for estimating bilirubin levels

ABSTRACT

Systems, methods, and devices are provided for estimating bilirubin levels. In one aspect, a method for estimating the level of bilirubin in a patient includes receiving image data for at least one image including a region of the patient&#39;s skin and a color calibration target. Color-balanced image data for the skin region is generated based on a subset of the image data corresponding to the color calibration target and the skin region. The bilirubin level in the patient is estimated based on the color-balanced image data for the skin region.

CROSS-REFERENCE

This application is a continuation-in-part application and claims thebenefits of U.S. Provisional Application No. 62/041,492, filed Aug. 25,2014 and PCT Application No. PCT/US2014/024761, filed Mar. 12, 2014,which claims the benefit of U.S. Provisional Application No. 61/777,097,filed Mar. 12, 2013; the entire disclosures of which are incorporatedherein by reference.

BACKGROUND

An estimated 60-84% of newborn infants develop neonatal jaundice, whichproduces a yellowing of the skin caused by the accumulation of excessbilirubin (hyperbilirubinemia), a naturally occurring compound producedby the breakdown of red blood cells. Although this condition istypically harmless and resolves within a few days, highly elevatedlevels of bilirubin can lead to kernicterus, a devastating andirreversible neurological condition characterized by deafness, cerebralpalsy, profound developmental delay, or even death.

Current approaches for monitoring bilirubin levels in infants typicallyrequire repeated testing in a hospital setting. The blood concentrationof bilirubin can be determined by the total serum bilirubin (TSB)measured from a blood sample, or via a transcutaneous bilirubinometer(TcB) measurement accomplished using a non-invasive but costlyinstrument. These tests are often unavailable in resource-poor settings,thus impeding early detection and treatment of kernicterus. Visualassessments, which are frequently used as an alternative to these tests,are often inaccurate and can be confounded by factors such as lightingor skin tone. Accordingly, improved approaches are needed for providingnon-invasive, cost-effective screening for excessive bilirubin levels.

SUMMARY

Systems, methods, and devices are provided for estimating bilirubinlevels. In many embodiments, a mobile device is used to capture imagedata of a patient's skin and a color calibration target. The image datais processed to generate an estimation of the bilirubin level. The imageprocessing can include transforming the image data into a plurality ofdifferent color spaces to facilitate assessment of the overallyellowness of the skin while compensating for color differences causedby lighting, skin tone, and other potentially confounding factors. Thescreening techniques described herein can be practiced by users (e.g.,parents, medical professionals, community health workers) in anoutpatient setting (e.g., a patient's home) without requiringspecialized medical equipment, thereby improving the convenience,accessibility, and cost-effectiveness of bilirubin monitoring.

Thus, in a first aspect, a method is provided for estimating the levelof bilirubin in a patient. The method includes receiving image data forat least one image including a region of the patient's skin and a colorcalibration target. Color-balanced image data for the skin region isgenerated based on a subset of the image data corresponding to the colorcalibration target and the skin region. The bilirubin level in thepatient is estimated based on the color-balanced image data for the skinregion. In many embodiments, the image data can be obtained with anysuitable imaging device. The imaging device can collect image dataindependently of additional attachments or equipment, such as externallenses, filters, or other specialized hardware.

The bilirubin level can be estimated using only image data of thepatient's skin color at a particular point in time, or by comparing theskin color image data to baseline skin color image data. For example,the method can further include receiving baseline skin color data forthe patient corresponding to when the patient has a reference bilirubinlevel (e.g., approximately zero when baseline data is obtained within 24hours of birth). The bilirubin level can be estimated based on one ormore differences between the baseline skin color data and thecolor-balanced image data for the skin region. The baseline skin colordata for the patient can be generated by capturing baseline image datafor the patient when the patient has the reference bilirubin level. Thebaseline image data can correspond to at least one image including theskin region and a baseline color calibration target. Color-balancedbaseline image data for the skin region can be generated based on asubset of the baseline image data corresponding to the baseline colorcalibration target and the skin region. The baseline skin color data canbe generated based on the color-balanced baseline image data for theskin region.

A standardized color calibration target can be used to facilitate thecolor balancing process. The color calibration target can include aplurality of standardized color regions, including a white color region.The standardized color regions can include a black region, a grayregion, a light brown region, a cyan region, a magenta region, a yellowregion, and a dark brown region. The color calibration target can atleast partially define an opening configured to expose the skin regionto permit capturing of image data for the skin region. The standardizedcolor regions can be disposed in a known arrangement surrounding theopening. Accordingly, the process for generating color-balanced imagedata for the skin region can include processing the received image datato identify a subset of the image data corresponding to the exposed skinregion and a subset of the image data corresponding to the white colorregion. The white color region data can be processed to determineobserved color values for the white color region. Color-balanced imagedata for the exposed skin region can be generated based on the observedcolor values for the white color region. The observed color values forthe white color region can include any suitable color space values, suchas red, green, blue (RGB) color space values.

The image data can be transformed into a plurality of different colorspaces in order to detect yellow discoloration of the skin. For example,the color-balanced image data for the skin region can include RGB colorspace data, and a method of estimating the level of bilirubin in apatient can further include transforming the RGB color space data intoat least one other color space to generate color-balanced image data forthe exposed skin region for the at least one other color space. The atleast one color space can include: (a) a cyan, magenta, yellow, andblack (CMYK) color space; (b) a YCbCr color space; and/or (c) a Labcolor space.

A plurality of chromatic and achromatic features can be generated basedon the image data. In some instances, the received image data caninclude an image obtained using flash illumination and an image obtainedwithout using flash illumination. Estimating the bilirubin level caninclude processing a plurality of normalized chromatic and achromaticfeatures to select a first estimated range of the bilirubin level fromone of a plurality of different bilirubin ranges. The features can beprocessed using an approach based on the selected first estimated rangeof the bilirubin level to generate a final estimate of the bilirubinlevel. The plurality of different bilirubin ranges can include a lowrange, a medium range, and a high range. The plurality of features caninclude selected color values of the skin region for a plurality ofdifferent color spaces. In some instances, the plurality of features caninclude a calculation of a color gradient across the skin region.

Estimation of the bilirubin level can involve performing one or moreregressions. For example, processing the features to select a firstestimated range of the bilirubin level can include performing a seriesof regressions, including at least one of: (a) a linear regression, (b)an encapsulated k-Nearest Neighbor regression, (c) a lasso regression,(d) a LARS regression, (e) an elastic net regression, (f) a supportvector regression using a linear kernel, (g) a support vector regressionassigning higher weight to higher-rated bilirubin values, and/or (h) arandom forest regression. Processing the features to select a firstestimated range of the bilirubin levels can include performing a seriesof regressions, including: (a) a linear regression, (b) an encapsulatedk-Nearest Neighbor regression, (c) a lasso regression, (d) a LARSregression, (e) an elastic net regression, (f) a support vectorregression using a linear kernel, (g) a support vector regressionassigning higher weight to higher-rated bilirubin values, and (h) arandom forest regression. Using a processing approach based on theselected first estimated range of the bilirubin level can includeperforming a final random forest regression that uses the plurality ofnormalized chromatic and achromatic features and the selected firstestimated range of the bilirubin level as the features for the finalrandom forest regression. In some instances, estimating the bilirubinlevel includes determining color space value for the patient's skin andusing a processing approach based on the determined color space value toestimate the bilirubin level. The regression equations can also includefeatures from the baseline image (e.g., color space values from one ormore color spaces).

In another aspect, a mobile device configured to estimate the level ofbilirubin in a patient is provided. The device includes a cameraoperable to capture image data for a field of view, a processoroperatively coupled with the camera, and a data storage deviceoperatively coupled with the processor. The data storage device canstore instructions that, when executed by the processor, cause theprocessor to receive image data for an image captured by the camera, theimage including a region of the patient's skin and a color calibrationtarget. The instructions can cause the processor to generatecolor-balanced image data for the skin region based on a subset of theimage data corresponding to the color calibration target and the skinregion, and estimate the bilirubin level in the patient based on thecolor-balanced image data for the skin region. In many embodiments, themobile device can be used to estimate the bilirubin level independentlyof any external attachments to the mobile device (e.g., lenses, filters)or any other specialized mobile device equipment.

The color calibration target can at least partially define an openingconfigured to expose the skin region to permit capturing of image datafor the skin region, and can include a plurality of standardized colorregions including a white color region. The instructions can cause theprocessor to process the received image data to identify a subset of theimage data corresponding to the exposed skin region and a subset of theimage data corresponding to the white color region. The white colorregion data can be processed to determine observed color values for thewhite color region. Color-balanced RGB image data can be generated forthe exposed skin region based on the observed color values for the whitecolor region. Color-balanced image data for the exposed skin region canbe generated for at least one other color space by transforming thecolor-balanced RGB image data into the at least one other color space. Aplurality of normalized chromatic and achromatic features can beprocessed to select a first estimated range of the bilirubin level fromone of a plurality of different bilirubin ranges. The features can beprocessed using an approach based on the selected first estimated rangeof the bilirubin level to generate a final estimate of the bilirubinlevel.

Alternatively or in addition, the color-balanced image data for theexposed skin region can be processed to determine a color space valuefor the patient's skin. A plurality of normalized chromatic andachromatic features can be processed using an approach based on thedetermined patient's skin color space value to estimate the bilirubinlevel.

The mobile devices described herein can further include a flash unitoperable to selectively illuminate the field of view. The received imagedata processed to estimate the bilirubin level can include an imagecaptured with the field of view being illuminated by the flash unit andan image captured with the field of view not being illuminated by theflash unit.

In another aspect, a method for estimating the level of bilirubin in apatient is provided. The method includes receiving, from a mobiledevice, image data for an image including a skin region of a patient anda color calibration target. Color-balanced image data for the skinregion can be generated based on a subset of the image datacorresponding to the color calibration target and the skin region, viaone or more processors. The bilirubin level in the patient can beestimated based on the color-balanced image data for the skin region,via the one or more processors. The estimated bilirubin level can betransmitted to the mobile device. In many embodiments, at least one ofreceiving the image data and transmitting the estimated bilirubin levelare performed using short message service (SMS) text messaging. Theimage data can be obtained by the mobile device without the use ofexternal attachments, hardware add-ons, or any other specializedequipment.

Other objects and features of the present invention will become apparentby a review of the specification, claims, and appended figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1A illustrates guidelines for the use of phototherapy to treatneonatal jaundice, in accordance with many embodiments;

FIG. 1B illustrates guidelines for the use of exchange transfusion totreat neonatal jaundice, in accordance with many embodiments;

FIG. 1C illustrates a Bhutani nomogram for assessing risk associatedwith neonatal jaundice, in accordance with many embodiments;

FIG. 2 illustrates the use of a mobile device to capture image data usedto estimate the bilirubin level in a patient, in accordance with manyembodiments;

FIG. 3A through FIG. 3D illustrate color calibration targets used inconjunction with a mobile device to capture image data used to estimatebilirubin level, in accordance with many embodiments;

FIG. 4A through FIG. 4C illustrate exemplary user interfaces of a mobileapplication for estimating bilirubin level, in accordance with manyembodiments;

FIG. 5A through FIG. 5F illustrate exemplary image data collected foruse in estimating bilirubin level, in accordance with many embodiments;

FIG. 6 illustrates a method for estimating the bilirubin level in apatient, in accordance with many embodiments;

FIG. 7 illustrates a method for generating color-balanced image data fora patient's skin, in accordance with many embodiments;

FIG. 8 illustrates identification of a standardized color region in animage, in accordance with many embodiments;

FIG. 9A illustrates a method for estimating the bilirubin level in apatient, in accordance with many embodiments;

FIG. 9B illustrates another example of a method for estimating thebilirubin level in a patient, in accordance with many embodiments;

FIG. 10 illustrates a method for estimating the bilirubin level in apatient, in accordance with many embodiments;

FIG. 11A illustrates a method for estimating the bilirubin level usingbaseline skin color data, in accordance with many embodiments;

FIG. 11B illustrates a method for generating baseline skin color data,in accordance with many embodiments;

FIG. 12 illustrates a mobile device for estimating bilirubin level, inaccordance with many embodiments; and

FIG. 13 illustrates a mobile device in a communication with a dataprocessing system for estimating bilirubin level, in accordance withmany embodiments.

DETAILED DESCRIPTION

The systems, devices, and methods described herein provide improvedapproaches for estimating the bilirubin level in a patient (e.g., aninfant or an adult). In many embodiments, the bilirubin level isestimated based on the visual characteristics of the patient's skin asdetermined from image data (e.g., pictures, videos). Color balancing canbe applied to compensate for variations in the image data due todifferent lighting conditions. For example, the systems, methods, anddevices described herein can be used to color balance images of thepatient's skin, extract intensities of various reflected wavelengths andother chromatic and/or achromatic properties from the skin, and estimatebilirubin levels (e.g., using machine learning or other suitablealgorithms).

In many embodiments, a mobile device configured with suitable softwarecan be used to capture images of a patient's skin and a standardizedcolor calibration target. The image data of the skin and target can beused to generate color-balanced image data that is analyzed to estimatethe bilirubin level in the patient. For example, the color-balancedimage data can be transformed into a plurality of different color spacesin order to extract features representative of the yellowness of theskin, and these features can be used in a plurality of regressions togenerate the bilirubin estimate.

Contrary to existing approaches for measuring bilirubin level, whicheither rely upon an invasive blood test or utilize costlyinstrumentation, the systems, devices, and methods described hereinenable convenient, portable, and inexpensive bilirubin level estimationthat can easily be performed by non-medical personnel using a personalmobile device, thereby improving the accessibility andcost-effectiveness of bilirubin monitoring. Advantageously, the methodsdescribed herein can be performed on a mobile device without requiringthe use of external attachments, hardware add-ons, or any otherspecialized mobile device equipment. Notably, the disclosed techniquesprovide accurate bilirubin level estimation over a large range ofbilirubin concentrations, in contrast to TcB measurement which exhibitsreduced accuracy at high bilirubin levels. Additionally, the methodsdescribed herein account for diversity in skin tones as well asdifferent lighting conditions, thereby improving the accuracy andflexibility of non-invasive bilirubin level estimation. Furthermore, theuse of mobile device software platforms enables easy and rapid updatingof the estimation methods and algorithms described herein, thus allowingimprovements and upgrades to be made as necessary.

Turning now to the drawings, FIG. 1A illustrates guidelines for the useof phototherapy to treat neonatal jaundice. Similarly, FIG. 1Billustrates guidelines for the use of exchange transfusion to treatneonatal jaundice. The guidelines can be used by a medical professionalto determine the appropriate course of treatment, based on the infant'sage, total serum bilirubin (TSB), number of weeks of gestation (e.g.,≧35 weeks), and other risk factors. Risk factors may include isoimmunehemolytic disease, G6PD deficiency, asphyxia, significant lethargy,temperature instability, sepsis, acidosis, or an albumin level lowerthan 3.0 grams per deciliter. The curves depicted in FIGS. 1A and 1Bindicate exemplary treatment thresholds for low risk, medium risk, andhigh risk infants.

FIG. 1C illustrates a Bhutani nomogram 110 for assessing risk associatedwith neonatal jaundice. The Bhutani nomogram 110 can be used by amedical professional to assess an infant's risk for developinghyperbilirubinemia based in the infant's postnatal age and bilirubinlevel. For example, the Bhutani nomogram 110 can include a plurality ofpercentile curves 112 used to define a low risk zone 114, a lowintermediate risk zone 116, a high intermediate risk zone 118, and ahigh risk zone 120.

FIG. 2 illustrates mobile device-based estimation of bilirubin levels,in accordance with many embodiments. An imaging device, such as a mobiledevice 202 (e.g., a smartphone, tablet) is used to capture an image of askin region 204 of a patient 206 and a color calibration target 208.Suitable skin regions for the approaches described herein include theforehead and sternum, as well as any other prominent, flat regions ofskin that are likely to be evenly lit. Additionally, since jaundicetypically first appears on the forehead and slowly progresses downwardon the body, regions closer to the forehead can potentially be moreinformative for diagnosis. Accordingly, in many embodiments, the colorcalibration target 208 is placed on the abdomen of the patient 206 nearthe sternum. The mobile device 202 includes a camera (not shown) forrecording image data, as well as a display 210 used for presenting auser interface (UI) of a mobile software application (“mobile app” or“app”) for estimating bilirubin levels based on the image data. Themobile device 202 can be a personal device of a user (e.g., a parent,medical professional, community health worker, etc.), such that the userneed only install the mobile app on their personal device in order toperform the methods described herein, with no other instrumentation orhardware being needed besides the color calibration target 208. Notably,the mobile device 202 can be used to practice the methods describedherein independently of any further attachment or accessory to themobile device, such as an external lens, filter, or other specializedmobile device equipment. Additional details on suitable software andhardware components for the mobile device 202 are provided in furtherdetail below.

FIG. 3A through FIG. 3D illustrate color calibration targets that can beused in conjunction with a mobile device for estimating bilirubinlevels, in accordance with many embodiments. The color calibrationtargets described herein (also known as “color calibration cards” and“color cards”) can be used to account for differences in lighting orother environmental conditions that affect the resultant color balanceof the skin image data. Referring to FIG. 3A, a color calibration target300 can be provided on a rectangular card 302. The card 302 can becardstock of any size suitable for placement onto the skin of a patient(e.g., an infant), such as approximately the size of a business card. Insome instances, the card 302 can be sterilizable or disposable, so as toprevent the spread of pathogens between patients. The color calibrationtarget 300 can include a plurality of standardized colored regions 304,which can be printed onto the card 302. The colored regions 304 can beof any suitable size, number, or shape (e.g., square, rectangular,polygonal, circular, elliptical, etc.). For example, the colorcalibration target 300 is depicted as including eight identically-sizedsquare colored regions 304 a-h. Each of the standardized colored regions304 can be of a different color (e.g., black, gray, white, cyan,magenta, yellow, light brown, dark brown) and be positioned in a knownarrangement on the card 302. For example, in the embodiment of FIG. 3A,304 a is a black region, 304 b is a gray region (e.g., 50% gray), 304 cis a white region, 304 d is a light brown region (e.g., a first skintone), 304 e is a cyan region, 304 f is a magenta region, 304 g is ayellow region, and 304 h is a dark brown region (e.g., a second skintone). Other arrangements and combinations of colors can also be used.Furthermore, the back side (not shown) of the card 302 can include oneor more adhesive regions enabling the card 302 to be removably attachedto the patient's skin. The back side can also include relevantinstructions, such as instructions for the user on how to download theaccompanying mobile app onto their personal mobile device.

FIG. 3B illustrates an alternative embodiment of a color calibrationtarget 320. The color calibration target 320 is substantially similar tothe color calibration target 300, except it also defines an opening 322.The opening 322 can constrain the calibration target 300 to lie flushwith the skin of the patient for more consistent lighting. The opening322 can be positioned such that when the color calibration target 300 isplaced on the patient, the skin region of interest is exposed throughthe opening 322. The opening 322 can be entirely defined (e.g.,completely surrounded) or partially defined (e.g., partially surrounded)by the target 320. A plurality of standardized color regions 324,including a white color region, can be positioned around the opening 322in a known arrangement. The embodiment of FIG. 3B further includes lightgray, medium gray, and dark gray color regions for a total of 10different standardized color regions. FIG. 3C illustrates a colorcalibration target 340 provided on a square card 342, with a pluralityof standardized color regions 344 surrounding a square opening 346. Someof the color regions 344 can include more than one color, such as colorregion 348, which includes a central black square bordered by white.FIG. 3D illustrates a color calibration target 360 having a plurality ofstandardized color regions 362 surrounding an opening 364. The opening364 can be adjacent to some or all of color regions 362. The colorregions 362 are depicted in FIG. 3D as a series of rectangular regionswith differing aspect ratios. The colors, geometry, and arrangement ofthe color regions of the color calibration targets described herein canbe selected based on the image processing and analysis methods to beused, such as the embodiments discussed below.

FIG. 4A through FIG. 4C illustrate exemplary user interfaces (UIs) of amobile app for estimating bilirubin level, in accordance with manyembodiments. FIG. 4A illustrates a summary UI 400 for displaying variousstatistics and metrics relating to a patient. The UI 400 can includepatient identification information 402 (e.g., a patient name, study IDnumber), time of birth 404, and any available bilirubin level results406 (e.g., TSB and/or TcB results). The UI 400 can also include a button408 or other interactive elements enabling the user to collect imagedata (e.g., a video sample or photographic sample) of a patient. In someinstances, video samples can be advantageous for eliminating issues ofmotion blur. FIG. 4B illustrates an instruction UI 420 for assisting auser with capturing image data of a patient. The UI 420 can includegraphical and/or textual instructions 422 directing the user to performthe appropriate steps. For example, the instructions 422 can instructthe user to place a color calibration target onto the abdomen of thepatient below the sternum. In many embodiments, the instructions 422 areadapted to be easily comprehended by individuals without medicaltraining, thereby allowing non-medical professionals to operate themobile app. FIG. 4C illustrates a live preview UI 440 displayed to auser during the image capture process. The UI 440 can include a videopreview 442 of the current field of view of the camera of the mobiledevice. The UI 440 can include one or more positioning targets 444 toassist the user in positioning the mobile device. For example, thepositioning target 444 can be a box, frame, or colored region overlaidonto the video preview 422 to indicate the appropriate placement of thecolor calibration target. The size and location of the positioningtarget 444 can be selected to constrain the distance of the mobiledevice from the patient to within an optimal range, and also to ensurethat the field of view adequately captures the appropriate skin regionsand the color calibration target. Once the field of view is correctlyaligned, the user can record image data by tapping a record button 446.Additional details regarding the image collection and analysis processare provided below.

FIG. 5A through FIG. 5F illustrate exemplary image data collected by themobile app, in accordance with many embodiments. Following the recordingprocess, the mobile app can display the image data to the user forreview and approval. Furthermore, in order to ensure that the capturedimages are of sufficient quality for processing and analysis, the mobileapp can include functionality for detecting the image quality, and alertthe user to images that do not meet a quality threshold and therebynecessitate a retake. FIG. 5A illustrates an image 500 of sufficientquality, in which the lighting is satisfactory and the patient and colorcalibration target are both clearly visible. FIG. 5B illustrates adeficient image 502, in which portions of the image are obstructed byglare. FIG. 5C illustrates a deficient image 504, in which the image istoo bright. FIG. 5D illustrates a deficient image 506, in which thecolor calibration target is partially obstructed. FIG. 5E illustrates adeficient image 508, in which the image is partially obscured byshadows. FIG. 5F illustrates a deficient image 510, in which the imageis too dark.

Alternatively, the app may automatically recognize and save images withsufficient quality, without manual input from the user. In manyembodiments, rather than checking the image quality after recording, theapp can check the live image feed from the camera in real time, andalert the user of any potential images prior to image collection.Optionally, the app can automatically determine and report the source ofimage quality issues, and offer suggestions to the user for how toadjust accordingly.

FIG. 6 illustrates a method 600 for estimating the bilirubin level in apatient, in accordance with many embodiments. The method 600, as withall other methods described herein, can be practiced using any of thesystems and devices disclosed herein, such as a mobile device or aseparate computing system (e.g., a remote server) in communication witha mobile device. Certain steps of the method 600, as with all othermethods described herein, can be optional, or may be combined with orsubstituted for suitable steps of other methods disclosed herein.

In act 610, image data is received for at least one image including aregion of the patient's skin and a color calibration target. The imagedata can be collected using the camera of a mobile device, as previouslydescribed herein. Image data can include photographic data (e.g., asingle image or a sequence of images), video data, or suitablecombinations thereof. Photographic data and video data can be capturedsequentially or simultaneously (e.g., images are taken during videorecording). The image data can be obtained with and/or without usingflash illumination. In some instances, flash illumination can be used tocancel out environmental lighting, such that the lighting in theresultant images is determined solely by contributions from the flashillumination. This can be advantageous to produce more consistentlighting or in situations where the environmental lighting is suboptimal(e.g., too dark, strongly colored, etc.).

In many embodiments, the image data is collected using a mobile app thatcontrols the mobile device's flash unit, as well as the sequence andnumber of images taken. For example, the app can turn on the flash unitduring the initial positioning of the mobile device, so that the usercan assess the amount of glare in the image (e.g., via the videopreview) and reposition the device as necessary to reduce or eliminateglare. During the recording process, the app can control the mobiledevice to first obtain image data with the flash unit on and then obtainimage data with the flash unit off, thereby generating two image setswith and without flash illumination, respectively. For example, when theuser initiates the image capture process (e.g., by pressing a recordbutton), the app can be configured to record a video of the patient andcalibration target, with the flash unit on for the first half and offfor the second half. The app can also capture two photographs takenduring the first and second halves of the video recording, respectively.The overall length of the video can be any suitable time, such asapproximately 10 seconds. Alternatively or in combination, the app canbe configured to capture a sequence of still images both with andwithout flash. The timing of the image capture process can further beconfigured to ensure that the image sensor (e.g., charge-coupled device(CCD) array) of the mobile device has stabilized before the next imageis taken. For example, the app can include a specified amount of delaytime (e.g., three to four seconds) before recording each image set.

As previously discussed, once the image capture sequence is complete,the mobile app can analyze the image data to determine whether it is ofsufficient quality for subsequent use. For instance, the app canimplement suitable image analysis techniques (e.g., computer vision) toassess image quality. An exemplary procedure involves extracting thecolor calibration target from the captured image data, checking thecolor consistency across each color region of the calibration target(e.g., determine whether the standard deviation of pixel values for eachcolor region falls below a predetermined threshold), and recommendingthat the user retake any images that do not pass this test. In someinstances, the app can be configured to capture multiple sets of imagedata per session, so as to maximize the likelihood that at least some ofthe image data will meet the quality standards. The approved images canthen be processed on the mobile device or transmitted to a separatecomputing system for processing, as described in further detail below.

In act 620, color-balanced image data for the skin region is generatedbased on a subset of the image data corresponding to the colorcalibration target and the skin region. Color balancing can be performedin order to compensate for different lighting conditions, as the colorcontent of the image data can vary considerably based on theenvironmental lighting (e.g., intensity, color, type (halogen,fluorescent, natural, etc.)). In many embodiments, the image data isinitially normalized by dividing the three red, green, blue (RGB) colorchannels by the overall luminescence of the image. Furthermore, theobserved pixel color values for one or more of the standardized colorregions of the color calibration target can be used to determine thecolor balancing adjustments to be applied to the skin regions in theimage data, as discussed below.

In act 630, the bilirubin level in the patient is estimated based on thecolor-balanced image data for the skin region. Since hyperbilirubinemiaproduces a yellow discoloration of the skin, the bilirubin level can bedetermined based on the amount of yellow in the color-balanced skinregion image data. This determination can be performed using anysuitable technique, such as by transforming the color-balanced imagedata into a plurality of different color spaces selected to facilitatequantifying the overall yellowness of the skin. Image features generatedfrom the transformed image data can be input into a series of machinelearning regressions designed to estimate the concentration of bilirubinin the patient's body. These approaches are discussed in further detailbelow.

FIG. 7 illustrates a method 700 for generating color-balanced imagedata, in accordance with many embodiments. The method 700 can be appliedto photographic and/or video data obtained of a patient's skin regionand color calibration target, as previously discussed. In act 710, thereceived image data is processed to identify the image datacorresponding to the exposed skin region and the image datacorresponding to the white color region. For example, the image data canbe segmented to extract the pixel values of the color regions of thecolor calibration target and the skin regions of interest (e.g.,sternum, forehead). With respect to the color calibration target, if themobile app UI includes a positioning target for aligning the calibrationtarget (e.g., positioning target 444 of FIG. 4C), then the approximatepixel coordinates of the color calibration target in the image data arealready known, and the search space for the color regions can beconstrained accordingly. The positioning of the color regions can bedetermined by identifying at least two color regions on the calibrationtarget and using these as a basis for extrapolating the positions of therest of the color regions.

FIG. 8 illustrates identification of a standardized color region in animage, in accordance with many embodiments. The color regionsegmentation process can be implemented via a suitable algorithmconfigured to apply predetermined thresholds to the image. In manyembodiments, since the cyan, magenta, and yellow color regions havedistinct hues and relatively high saturation, the algorithm initiallyattempts to identify at least two of these regions. The algorithm canconvert the image data to a hue, saturation, value (HSV) color space andapply empirically determined thresholds to the hue and saturationchannels, thereby obtaining a thresholded hue image 800 and thresholdedsaturation image 802. An “AND” operation can be performed on the twothresholded images to separate the specified color region from the restof the image, thereby producing a segmented image 804. Furthermore,since the approximate size of each color region is predetermined, thisinformation can be used to differentiate the color region from the noisein the image data (e.g., using edge detection, morphological operations,etc.). For example, the algorithm can utilize an opening operation andCanny edge detection. The algorithm can then use contour detection toidentify the boundary of the color region, with contour smoothingperformed using suitable techniques such as the Douglas-Peukeralgorithm. Since the overall arrangement of the color regions is known,once the positions of two color regions are determined, the overallorientation of the calibration target can be calculated and used toextrapolate the positions of the other regions, including the whitecolor region.

In act 720, the white color region data is processed to determineobserved color values for the white color region. In act 730,color-balanced image data for the exposed skin region is generated basedon the observed color values for the white color region. The observedcolor values for the white color region can include RGB color values,which can be used to adjust the RGB values of the image data for theskin region. For example, for observed color values for the white regionx=(R′, G′, B′), the adjusted color values for the skin region (R, G, B)can be obtained by

$\begin{bmatrix}R \\G \\B\end{bmatrix} = {\begin{bmatrix}{255/R_{w}^{\prime}} & 0 & 0 \\0 & {255/G_{w}^{\prime}} & 0 \\0 & 0 & {255/B_{w}^{\prime}}\end{bmatrix}\begin{bmatrix}R^{\prime} \\G^{\prime} \\B^{\prime}\end{bmatrix}}$

where K_(W)=(R_(W)′, G_(W)′, B_(W)′) is the raw observed color values ofthe white region on the color calibration target. The observed colorvalues for the white color region can then be used to perform whitebalancing of the image data.

Once color balancing has been performed, the image data can be processedto estimate the mean red, green, and blue values, and the gradients ofcolors in the skin region. Various color transformations can be employedto approximate properties in the skin such as hue, gamma, andsaturation. The extracted properties can then be used as features in astacked regression and classification algorithm, which results in afinal estimate of the bilirubin level.

FIG. 9A illustrates a method 900 for estimating the bilirubin level in apatient, in accordance with many embodiments. The method 900 can bepracticed using color-balanced RGB image data obtained as previouslydescribed herein. In act 910, the color-balanced RGB image data istransformed into at least one other color space to generatecolor-balanced image data for the exposed skin region for the at leastone other color space. As discussed above, the yellowness of thecolor-balanced skin region image data correlates to the bilirubin levelin the patient's body. In many embodiments, the image sensor (e.g., CCDor CMOS sensor) of the mobile device interpolates reflected light intored, green, and blue wavelengths, which can prevent the image sensorfrom capturing the reflection of the yellow wavelength band clearly.Accordingly, the color-balanced RGB data can be transformed into aplurality of different color spaces, such as cyan, magenta, and yellow(CMY); cyan, magenta, yellow, and black (CMYK); YCbCr; or Lab colorspaces. Any suitable number or combination of color spaces can be used.For example, in many embodiments, the RGB image data is transformed intoCMYK, YCbCr, and Lab color spaces. Alternatively, the RGB image data canbe transformed into CMY, YCbCr, and Lab color spaces.

Optionally, in addition to color transformations, a linear colorgradient can be used to calculate the change in color across theportions of the image corresponding to the skin region. For example, thegradient can be calculated by running a 3×3 Sobel gradient filter acrossthe color channel, and then averaging the outputs inside the portion.This can be performed in the red, blue, and green color planes, thusresulting in three additional features.

In act 920, a plurality of normalized chromatic and achromatic featuresare processed to select a first estimated range of the bilirubin levelfrom a plurality of different bilirubin ranges. The features can bechromatic and/or achromatic values or properties (e.g., luminescence,hue, gamma, saturation) for the skin region, with each featurecorresponding to a color space value of the color spaces used. In someinstances, the features can include calculations of one or more colorgradients across the skin the region. Any suitable number andcombination of features can be used. For example, three features can beextracted from each of four color spaces (e.g., RGB, CMY, YCbCr, Lab; orRGB, CMYK, YCbCr, Lab) to obtain 12 chromatic and achromatic features.Furthermore, features can be separately extracted from image dataobtained with flash illumination and without flash illumination,respectively, resulting in a total of 24 chromatic and achromaticfeatures. As another example, 12 color planes can be obtained from eachimage of the skin region (three planes from each of four color spaces).The mean and median can be used in calculating one feature value fromeach region, thus resulting in 24 features per skin region. These 24features can be calculated for each flash and non-flash image. Togetherwith the color gradient features, this results in a total of 24+24+6=54features.

The features can be normalized based on the overall luminescence of theimage, as previously described herein with respect to act 620 of themethod 600. Optionally, the features can be preprocessed to have unitvariance and zero mean. The extracted features can be used as inputs tomachine learning regressions used for estimating the bilirubin level.The regressions can utilize some or all of the extracted features, withthe optimal subset of features to be used selected based on machinelearning techniques. The machine learning regressions described hereincan be trained on suitable data sets, such as clinical patient data, andcan incorporate any suitable number and combination of parametric andnon-parametric regression models. Exemplary regressions suitable for usewith the methods described herein are provided below. In manyembodiments, the initially calculated features and the output of theselected regressors are used to classify the estimated bilirubin levelinto one of a plurality of different bilirubin ranges, such as low,medium, and high ranges.

In act 930, the features are processed using a processing approach basedon the selected first estimated range of the bilirubin level to generatea final estimate of the bilirubin level. Similar to the act 920, thenormalized chromatic and achromatic features can be used to inform oneor more machine learning regressions. The inputs to the regressions candiffer based on whether the first estimated range of the bilirubin levelis low, medium, or high, as provided in greater detail below. Forexample, the classification of the first estimated range can be used asinput to the machine learning regressions. The results of the initialregression performed in act 920 can also be used as input. This“two-tiered” approach to bilirubin estimation can be used to generatemore accurate estimation results compared to direct estimation.

FIG. 9B illustrate a method 950 for estimating the bilirubin level in apatient, in accordance with many embodiments. The method 950 can bepracticed in combination with the method 900 in order to obtain a moreaccurate estimate of the bilirubin level. Similar to the method 900, themethod 950 can be practiced using color-balanced RGB image data obtainedas previously described herein. In act 960, the color-balanced RGB imagedata is transformed into at least one other color space to generatecolor-balanced image data for the exposed skin region for the at leastone other color space, as discussed above with respect to act 910 of themethod 900.

In act 970, the color-balanced image data for the exposed skin region isprocessed to determine a color space value for the patient's skin. Thecolor space value for the patient's skin can be used to classify thepatient's skin color into one of a plurality of different skin colortypes, such as light-skinned, medium-skinned, and dark-skinned. The skincolor type can be related to the race and/or ethnicity of the patient.The skin color type can be determined based on the color values of theskin region and/or color calibration target obtained from thecolor-balanced RGB image data. For example, the skin region can becompared to one or more standardized color regions of the colorcalibration target (e.g., the first and second skin tone color regions)to determine a skin color type.

In act 980, a plurality of normalized chromatic and achromatic featuresare processed using an approach based on the determined skin color toestimate the bilirubin level. The normalized chromatic and achromaticfeatures can be extracted from the color-balanced image data for one ormore different color spaces, as previously described above with respectto the method 900. The features, along with the determined skin color,can be used as inputs to suitable machine learning regressions, similarto the act 930 of the method 900.

As discussed above and herein, a custom machine learning regression canbe used to estimate the bilirubin level. The regression algorithm canemploy several different types of regressions that are parametric,non-parametric, and mixed. The regression algorithm can include a firststep in which all of the features are used in each regression aimed atestimating the total bilirubin level, and a second step in which theoutputs of each regression are averaged, thus resulting in a singlevalue for the bilirubin level. In many embodiments, the output of theregression and the age of the patient are used to categorize thebilirubin value as low risk, intermediate low risk, intermediate highrisk or high risk using the Bhutani nomogram described herein.

FIG. 10 illustrates a method 1000 for estimating the bilirubin level ina patient, in accordance with many embodiments. The method 1000 includesobtaining images from a camera 1002, color balancing the image data1004, performing feature extraction from the color-balanced image data1006, performing machine learning regression based on the features 1008,and generating a bilirubin estimate 1010.

Image data can be obtained from the camera of a mobile device under thecontrol of a suitable mobile app (act 1002). Some of the image data canbe obtained with flash illumination and some of the image data can beobtained without flash illumination. Once the app has verified that theimages are of sufficient quality for processing and analysis, the imagedata can be color balanced (act 1004). The color balancing can involveidentification of the image data subsets corresponding to the colorcalibration target, and automatic segmentation of the standardized colorregions of the target (act 1012) using the thresholding methodspreviously described herein. The segmented white color region can beused to white balance the image data (act 1014), thereby generatingcolor-balanced image data. The color-balanced image data can be RGBimage data. One or more features can be extracted from thecolor-balanced image data (act 1006). The feature extraction process caninvolve transforming the image data from an RGB color space to aplurality of different color spaces (act 1016), as previously describedherein. The transformed image data can then be used to calculate aplurality of normalized chromatic and achromatic features (act 1018).

Some or all of the extracted features can be used as inputs to machinelearning regressions (act 1008). Any suitable number and/or combinationof regressions can be used. For example, an initial set of machinelearning regressions can include five different sets of regressions(acts 1020, 1022, 1024, 1026, and 1028). For example, the firstregression can include one or more encapsulated k-Nearest Neighbor (kNN)regressions (e.g., with k=7) (act 1020). This regression can utilize adatabase of known features and bilirubin values. When an unknown testvector is analyzed, the k-nearest neighbors can be found around the testvector in the database of features. A number of different distancemetrics can be used to calculate the nearest neighbors, including the L1and L2 norms. In many embodiments, a custom distance metric that cubesthe differences between the samples and sums them together can is alsoused. Feature points from the neighbors can then be used in a linearregression (e.g., a linear support vector regression). A new regressioncan be built each time that a new test point is analyzed. The parametersfor finding the nearest neighbors can be normalized values of luminosity(e.g., from the YCbCr color space transform) and the “green” or “red”channel (e.g., from the RGB color space). This can be used to guaranteethat the nearest neighbor calculation only occurs in two dimensions,thus ensuring the number of points in the nearest neighbor is tractable(e.g., approximately four to six points).

The second regression can include one or more lasso regressions and/orone or more least angle regressions (LARS) (act 1022). LARS can behelpful for deciding which features out of the total set of extractedfeatures are the most useful, using a variant of forward featureselection. For example, the best predictor(s) from the feature set canbe chosen by developing a single-feature, linear regression from eachfeature. The most correlated output can be chosen as the “first”feature. This prediction can be subtracted from the output to obtain theresiduals. The algorithm can diverge from other forward featureselection algorithms in that it attempts to find another feature withroughly the same correlation to the residuals as the first feature tothe output. It can then find the “equiangular” direction between the twoestimates, and can find a third feature that maximizes correlation tothe new residuals along the equiangular direction. A new angle can thenbe found from the previous features and a new feature added to the set.Features can be added in this way until the desired accuracy is met.

The third regression can include one or more elastic net regressions,also known as elastic net algorithms (act 1024). The elastic netregression is a combination of Lasso regression (highly related to LARSfor forward feature selection) and ridge regression (which uses an L2regularization). Instead of just using forward feature selection,however, the algorithm can also employ the L1 and L2 norms in itsobjective function. This makes it related to LARS and Lasso, but withcertain “backoff” regularization so that it can become more stable. Theparameters can be cross-validated using a stepwise exhaustive search.

The fourth regression can include one or more support vector (SV)regressions (act 1026). Two SV regressions can be employed in order tocapture the possible non-linear relationship between the image data andthe bilirubin levels. SV regression can be used to find a linearregression function in a high dimensional feature space. Then, the inputdata can be mapped into the space using a potentially nonlinearfunction. The first SV regression can uses a linear kernel and thesecond SV regression can assign higher weight to higher-rated bilirubinvalues using a nonlinear radial or sigmoidal basis function.

The fifth regression can include one or more random forest regressions(act 1028). For example, the fifth regression can use a random forestregression with 75 or 500 trees. A random forest is a collection ofestimators. It can use many “classifying” decision trees on varioussub-samples of the dataset. The outputs of these trees can be averagedto improve the predictive accuracy and to control over-fitting. Eachtree can be created using a random sub-sample (with replacement).

In many embodiments, other types of regressions can also be used inaddition to or substituted for one or more of the regressions describedherein. For example, one or more linear regressions can be also used.The method 1000 can be practiced using any suitable type of regression,including linear and non-linear regressions.

Following the initial regressions, one or more multi-layer classifierscan be used (act 1030). For example, the initially calculated features,along with the output of the initial set of regressors, can be used toclassify a first estimated range of the bilirubin level into one of aplurality of different bilirubin ranges, as previously described herein.The classifiers can include a random forest classifier, a support vectormachine, and a k-Nearest Neighbor (k=3). The results of all theclassifiers, as well as the log-likelihood for each class, can be usedas the features for a final stacked regression, which can be a finalrandom forest regression (act 1032). The original extracted features andthe results of the initial regressions can also be used as the featuresfor this final stage regression. The final regressor can be trainedusing suitable machine learning algorithms, such as AdaBoost. The finalregressor can be used as the final estimate of the bilirubin level 1010(e.g., measured in milligrams per deciliter). In order to avoidoverfitting, leave-one-out cross validation can be used at all levels oflearning.

In alternative embodiments, the final regression output can be the meanof the different regressions performed. In such embodiments, acts 1030and 1032 would be omitted and the outputs of the acts 1020, 1022, 1024,1026, and 1028 would be averaged to obtain the final bilirubin estimate1010.

FIG. 11A illustrates a method 1100 for estimating the bilirubin level ina patient using baseline skin color data, in accordance with manyembodiments. The method 1100 is similar to the method 600, except thatthe current skin color data of the patient is compared to baseline skincolor data in order to generate the bilirubin estimate. This approachcan be used to compensate for factors that may confound the imageanalysis, such as differing skin tones due to the patient's race orethnicity.

In act 1110, baseline skin color data for the patient is received, thebaseline skin color data corresponding to when the patient has areference bilirubin level. The reference bilirubin level can be a knownbilirubin level (e.g., based on TSB or TcB testing). If the patient isan infant, the baseline skin color data can be collected within thefirst 24 hours of the infant's birth, when the bilirubin level istypically very low, such as approximately zero. Suitable methods forcollecting and generating baseline skin color data are described below.

In act 1120, image data is received for at least one image including aregion of the patient's skin and a color calibration target. In act1130, color-balanced image data for the skin region is generated basedon a subset of the image data corresponding to the color calibrationtarget and the skin region. The image data collection andcolor-balancing processes can be similar to those previously describedherein with respect to acts 610 and 620 of the method 600, respectively.The acts 1120 and 1130 can be performed at any time after the baselineskin color data is received and can be repeated over any suitable periodof time (e.g., the first four to five days of the infant's life) so asto generate sequential image data sets used to determine whether theskin is becoming more yellow.

In act 1140, the bilirubin level in the patient is estimated based ondifferences between the baseline skin color data and the color-balancedimage data for the skin region. In many embodiments, the baseline skincolor data serves as a standard against which the current image data iscompared. As previously described herein with respect to act 630 of themethod 600, the estimation can be performed by transforming thecolor-balanced image data into a plurality of color spaces, extractingfeatures from the transformed image data, and then using the featuresfor machine learning regressions to generate a bilirubin estimate. Atleast some of the regressions described herein can also use some or allof the features extracted from the baseline skin color data.

FIG. 11B illustrates a method 1150 for generating baseline skin colordata, in accordance with many embodiments. In act 1160, baseline imagedata for the patient is captured when the patient has the referencebilirubin level, the baseline image data corresponding to at least oneimage including the skin region and a baseline color calibration target.The collection of the baseline image data can be similar to thecollection procedures previously described herein with respect to act610 of the method 600. The baseline color calibration target can be thesame as the color calibration target used for collecting regular imagedata, or can be a different color calibration target. Similarly, thebaseline image data can include the same skin regions as regular imagedata, or different skin regions. As described above, the referencebilirubin level can be any known bilirubin level, such as a bilirubinlevel determined by testing or taken within 24 hours of birth.

In act 1170, color-balanced baseline image data is generated for theskin region based on the baseline image data. The generation of thecolor-balanced baseline image data can be performed using any of thetechniques previously described herein with respect to regular imagedata. In act 1180, the baseline skin color data is generated based onthe color-balanced baseline image data for the skin region. This processcan involve feature extraction from the color-balanced baseline imagedata and using the features for machine learning regressions, asdiscussed above.

FIG. 12 illustrates a mobile device 1200 for estimating bilirubin levelin a patient, in accordance with many embodiments. The mobile device1200 includes a camera 1202 suitable for capturing image data, and aflash unit 1204 suitable for providing flash illumination. The camera1202 and flash unit 1204 can be built-in hardware of the mobile device1200. Alternatively, the camera 1202 and flash unit 1204 can be providedseparately from and coupled to the mobile device 1200 (e.g., via wiredor wireless communication). In some instances, the mobile apps describedherein can detect whether the camera 1202 is capable of capturing imageswith sufficiently high resolution for the subsequent image analysis, andcan alert the user if this criterion is not met.

The mobile device 1200 also includes an input unit 1206 for receivinginput from a user and a display 1208 for displaying content to the user.The input unit 1206 can include keyboards, mice, touchscreens,joysticks, and the like. The input unit 1206 can also be configured toaccept voice commands or gestural commands. The display 1208 can includea monitor, screen, touchscreen, and the like. In some instances, theinput unit 1206 and the display 1208 can be implemented across sharedhardware (e.g., the same touchscreen is used to accept input and displayoutput). As previously described, the display 1208 can display one ormore suitable Uls to the user, such as Uls of a mobile app forestimating bilirubin levels.

The mobile device 1200 includes one or more processors 1210, a memory orother data storage device 1212 storing image data as well as one or moresoftware modules 1214, and a communication unit 1216. The processors1210 can be operably coupled to the camera 1202 and/or flash unit 1204to control one or more functions (e.g., record function, zoom function,flash illumination function). The processor 1210 can also be operablycoupled to the memory 1212 such that the processor 1210 can receive andexecute instructions provided by the software module 1214. The softwaremodule 1214 can be implemented as part of the mobile apps describedherein and can provide instructions for carrying out one or more acts ofthe previously discussed methods. For example, the software module 1214can enable the mobile device 1200 to capture image data of a patient andcalibration target. In some instances, the software module 1214 canperform some or all of the image processing and analysis tasks disclosedabove (e.g., color balancing, feature extraction, machine learningregression). The software module 1214 can be adapted to a plurality ofdifferent types of mobile devices. Furthermore, the mobile device 1200can be configured to receive and install software updates, such asupdates improving one or more image capture, processing, and analysisalgorithms, thereby enabling the mobile app to be easily and quicklyupgraded as necessary.

The communication unit 1216 of the mobile device 1200 can be configuredto receive and/or transmit data (e.g., image data, bilirubin estimates,software updates, etc.) between the mobile device 1200 and a separatedevice or system, such as a remote server or other computing system. Thecommunication unit can use any suitable combination of wired or wirelesscommunication methods, including Wi-Fi communication. In some instances,the communication between the mobile device 1200 and the separate devicecan be performed using short message service (SMS) text messaging. Thecommunication unit 1216 can also be operably coupled to the processors1210, such that data communication to and from the mobile device 1200can be controlled based on instructions provided by the software module1214.

FIG. 13 illustrates a mobile device 1300 in communication with a dataprocessing system 1302 for estimating bilirubin levels, in accordancewith many embodiments. The mobile device 1300 can include any of thecomponents previously described herein with respect to the mobile device1200 of FIG. 12. The components of the data processing system 1302 canbe implemented across any suitable combination of physical and/orvirtualized computing resources (e.g., virtual machines), includingdistributed computing resources (“in the cloud”). In many embodiments,the data processing system 1302 is a remote server configured tocommunicate with a plurality of user mobile devices including the mobiledevice 1300. The communication can utilize any suitable wired orwireless communication methods, as described above.

The data processing system 1302 includes one or more processors 1304, amemory or other data storage device 1306 storing one or more softwaremodules 1308, and a communication unit 1310. The communication unit 1310can be used to communicate data (e.g., image data, bilirubin estimates,software updates, etc.) between the system 1302 and the mobile device1300 (e.g., via SMS text messaging). For example, the communication unit1310 can receive image data provided by the mobile device 1300, such asimage data that has not yet been color balanced. The data obtained fromthe mobile device 1300 can be stored in the memory 1306. The softwaremodule 1308 can provide instructions executable by the processors 1304to process and analyze the image data (e.g., color balancing, featureextraction, machine learning regressions), such as by performing one ormore acts of the methods described herein. The processors 1304 canoutput an estimate of the patient's bilirubin level, which can be storedin the memory 1306 and/or transmitted to the mobile device 1300. In someinstances, depending on user preference, the results can also betransmitted to a third party, such as a medical professional who canreview the results and provide the user with further instructions asnecessary. Optionally, based on the results, the processors 1304 canalso be configured to generate and display a recommended course ofaction to a user.

In an alternative embodiment, the mobile devices described herein can beconfigured to illuminate a patient's skin with different wavelengths oflight (e.g., 460 nm, 540 nm) and capture images of the illuminated skinusing a camera. The timing and sequence of illumination can becontrolled by a mobile app. The mobile app can analyze the collectedimage data to measure the intensity of different wavelengths of lightreflected from the skin regions. In many embodiments, the absorption ofdifferent wavelengths differs based on the color of the skin, includingyellow discoloration. Some wavelengths can be affected by bilirubinlevels, such that their intensities provide an indication of the amountof bilirubin in the patient's body. Accordingly, suitable machinelearning regressions and/or models can be developed to enable wavelengthabsorption to be used as input for estimating bilirubin levels in apatient. Advantageously, this approach can be more robust to differentenvironmental and situational conditions.

In many embodiments, the mobile device includes a front facing camera(e.g., a camera disposed on the same side of the mobile device as thescreen) can be used to capture image data (e.g., photographs, videos)that is used to assess ambient lighting conditions. Such data providesalternative and/or additional ambient lighting information that can beused to normalize the color data of the patient's skin during estimationof the bilirubin level of the patient.

One of ordinary skill in the art would appreciate that any of themethods presented herein can be performed using a mobile device incombination with a data processing system (“server-connected”implementation) or using the mobile device alone (“stand-alone”implementation), as desired. A server-connected implementation may beadvantageous for maintaining tighter control over how the system runsthe estimation algorithm, as well as ensuring that the system is usingthe most up-to-date version of the algorithm. A stand-aloneimplementation may be advantageous in terms of allowing for the use ofthe estimation algorithm when Internet connectivity and/or cell coverageis incomplete or inadequate, e.g., in low resource settings.

Exemplary experimental bilirubin estimation results obtained inaccordance with the methods herein are described in U.S. ProvisionalApplication No. 62/041,492, which is incorporated herein by reference.

The various techniques described herein may be partially or fullyimplemented using code that is storable upon storage media and computerreadable media, and executable by one or more processors of a computersystem. Storage media and computer readable media for containing code,or portions of code, can include any appropriate media known or used inthe art, including storage media and communication media, such as butnot limited to volatile and non-volatile, removable and non-removablemedia implemented in any method or technology for storage and/ortransmission of information such as computer readable instructions, datastructures, program modules, or other data, including RAM, ROM, EEPROM,flash memory or other memory technology, CD-ROM, digital versatile disk(DVD) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, solid statedrives (SSD) or other solid state storage devices, or any other mediumwhich can be used to store the desired information and which can beaccessed by the a system device. Based on the disclosure and teachingsprovided herein, a person of ordinary skill in the art will appreciateother ways and/or methods to implement the various embodiments.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. A method of estimating the level of bilirubin ina patient, the method comprising: receiving image data for at least oneimage including a region of the patient's skin and a color calibrationtarget; generating color-balanced image data for the skin region basedon a subset of the image data corresponding to the color calibrationtarget and the skin region; and estimating the bilirubin level in thepatient based on the color-balanced image data for the skin region. 2.The method of claim 1, further comprising receiving baseline skin colordata for the patient corresponding to when the patient has a referencebilirubin level, and wherein said estimating the bilirubin level isbased on one or more differences between the baseline skin color dataand the color-balanced image data for the skin region.
 3. The method ofclaim 2, wherein the baseline skin color data for the patient isgenerated by: capturing baseline image data for the patient when thepatient has the reference bilirubin level, the baseline image datacorresponding to at least one image including the skin region and abaseline color calibration target; generating color-balanced baselineimage data for the skin region based on a subset of the baseline imagedata corresponding to the baseline color calibration target and the skinregion; and generating the baseline skin color data based on thecolor-balanced baseline image data for the skin region.
 4. The method ofclaim 1, wherein the color calibration target comprises a plurality ofstandardized color regions including a white color region.
 5. The methodof claim 4, wherein the standardized color regions include a blackregion, a gray region, a light brown region, a cyan region, a magentaregion, a yellow region, and a dark brown region.
 6. The method of claim4, wherein the color calibration target at least partially defines anopening configured to expose the skin region to permit capturing ofimage data for the skin region.
 7. The method of claim 6, wherein thestandardized color regions are disposed in a known arrangementsurrounding the opening.
 8. The method of claim 6, wherein saidgenerating color-balanced image data for the skin region comprises:processing the received image data to identify a subset of the imagedata corresponding to the exposed skin region and a subset of the imagedata corresponding to the white color region; processing the white colorregion data to determine observed color values for the white colorregion; and generating color-balanced image data for the exposed skinregion based on the observed color values for the white color region. 9.The method of claim 8, wherein the observed color values for the whitecolor region comprise red, green, blue (RGB) color space values.
 10. Themethod of claim 1, wherein the color-balanced image data for the skinregion comprises RGB color space data, and the method further comprisestransforming the RGB color space data into at least one other colorspace to generate color-balanced image data for the exposed skin regionfor the at least one other color space.
 11. The method of claim 10,wherein the at least one other color space includes: (a) a cyan,magenta, yellow, and black (CMYK) color space; (b) a YCbCr color space;or (c) a Lab color space.
 12. The method of claim 10, wherein the atleast one other color space include: (a) a cyan, magenta, yellow (CMY)color space; (b) a YCbCr color space; or (c) a Lab color space.
 13. Themethod of claim 11, wherein the received image data includes an imageobtained using flash illumination and an image obtained without usingflash illumination.
 14. The method of claim 13, wherein said estimatingthe bilirubin level comprises: processing a plurality of normalizedchromatic and achromatic features to select a first estimated range ofthe bilirubin level from one of a plurality of different bilirubinranges; and processing the features using an approach based on theselected first estimated range of the bilirubin level to generate afinal estimate of the bilirubin level.
 15. The method of claim 14,wherein the plurality of different bilirubin ranges includes a lowrange, a medium range, and a high range.
 16. The method of claim 14,wherein the plurality of features comprises selected color values of theskin region for a plurality of different color spaces.
 17. The method ofclaim 14, wherein the plurality of features comprise a calculation of acolor gradient across the skin region.
 18. The method of claim 14,wherein processing the features to select a first estimated range of thebilirubin level comprises performing a series of regressions includingat least one of: (a) a linear regression, (b) an encapsulated k-NearestNeighbor regression, (c) a lasso regression, (d) a LARS regression, (e)an elastic net regression, (f) a support vector regression using alinear kernel, (g) a support vector regression assigning higher weightto higher-rated bilirubin values, and (h) a random forest regression.19. The method of claim 14, wherein processing the features to select afirst estimated range of the bilirubin level comprises performing aseries of regressions including: (a) a linear regression, anencapsulated k-Nearest Neighbor regression, (c) a lasso regression, (d)a LARS regression, (e) an elastic net regression, (f) a support vectorregression using a linear kernel, (g) a support vector regressionassigning higher weight to higher-rated bilirubin values, and (h) arandom forest regression.
 20. The method of claim 14, wherein said usinga processing approach based on the selected first estimated range of thebilirubin level comprises performing a final random forest regressionthat uses the plurality of normalized chromatic and achromatic featuresand the selected first estimated range of the bilirubin level as thefeatures for the final random forest regression.
 21. The method of claim1, wherein said estimating the bilirubin level comprises determining acolor space value for the patient's skin and using a processing approachbased on the determined patient's skin color space value to estimate thebilirubin level.
 22. A mobile device configured to estimate the level ofbilirubin in a patient, the device comprising: a camera operable tocapture image data for a field of view; a processor operatively coupledwith the camera; and a data storage device operatively coupled with theprocessor and storing instructions that, when executed by the processor,cause the processor to: receive image data for an image captured by thecamera, the image including a region of the patient's skin and a colorcalibration target; generate color-balanced image data for the skinregion based on a subset of the image data corresponding to the colorcalibration target and the skin region; and estimate the bilirubin levelin the patient based on the color-balanced image data for the skinregion.
 23. The mobile device of claim 22, wherein the color calibrationtarget at least partially defines an opening configured to expose theskin region to permit capturing image data for the skin region andincludes a plurality of standardized color regions including a whitecolor region, and wherein the instructions cause the processor to:process the received image data to identify a subset of the image datacorresponding to the exposed skin region and a subset of the image datacorresponding to the white color region; process the white color regiondata to determine observed color values for the white color region;generate color-balanced RGB image data for the exposed skin region basedon the observed color values for the white color region; generatecolor-balanced image data for the exposed skin region for at least oneother color space by transforming the color-balanced RGB image data intothe at least one other color space; process a plurality of normalizedchromatic and achromatic features to select a first estimated range ofthe bilirubin level from one of a plurality of different bilirubinranges; and process the features using an approach based on the selectedfirst estimated range of the bilirubin level to generate a finalestimate of the bilirubin level.
 24. The mobile device of claim 22,wherein the color calibration target at least partially defines anopening configured to expose the skin region to permit capturing imagedata for the skin region and includes a plurality of standardized colorregions including a white color region, and wherein the instructionscause the processor to: process the received image data to identify asubset of the image data corresponding to the exposed skin region and asubset of the image data corresponding to the white color region;process the white color region data to determine observed color valuesfor the white color region; generate color-balanced RGB image data forthe exposed skin region based on the observed color values for the whitecolor region; generate color-balanced image data for the exposed skinregion for at least one other color space by transforming thecolor-balanced RGB image data into the at least one other color space;process the color-balanced image data for the exposed skin region todetermine a skin color for the patient; and process a plurality ofnormalized chromatic and achromatic features using an approach based onthe determined skin color to estimate the bilirubin level.
 25. Themobile device of either one of claim 23 and claim 24, further comprisinga flash unit operable to selectively illuminate the field of view, thereceived image data processed to estimate the bilirubin level includingan image captured with the field of view being illuminated by the flashunit and an image captured with the field of view not being illuminatedby the flash unit.
 26. A method for estimating the level of bilirubin ina patient, the method comprising: receiving, from a mobile device, imagedata for an image including a skin region of a patient and a colorcalibration target; generating, via one or more processors,color-balanced image data for the skin region based on a subset of theimage data corresponding to the color calibration target and the skinregion; estimating, via the one or more processors, the bilirubin levelin the patient based on the color-balanced image data for the skinregion; and transmitting the estimated bilirubin level to the mobiledevice.
 27. The method of claim 25, wherein at least one of receivingthe image data and transmitting the estimated bilirubin level areperformed using short message service (SMS) text messaging.